{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DeepStream Launchpad Demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welcome to the DeepStream Launchpad Demo! In this Jupyter Notebook, you will learn how to build a Python application by constructing a Gstreamer pipeline using DeepStream elements and attaching a probe function to access meta data.\n",
    "\n",
    "The diagram below describes the workflow of a typical DeepStream application."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![workflow](ds_workflow.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Application Structure\n",
    "The application consists primarily of the following parts:\n",
    "1. Constructing and running **the pipeline**\n",
    "2. **probe function**, to extract metadata from the buffer at some pipeline element\n",
    "\n",
    "Some helper functions will also be described in this demo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Pipeline \n",
    "This demonstration employs the following pipeline**:\n",
    "\n",
    "![pipeline](ds_launchpad_pipeline.png)\n",
    "\n",
    "The pipeline elements are described here:\n",
    "* [`Gst-uridecodebin`](https://gstreamer.freedesktop.org/documentation/playback/uridecodebin.html?gi-language=python) - decode data from URI source into raw media\n",
    "* [`Gst-nvstreammux`](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvstreammux.html) - stream muxer plugin, to form a batch of buffers from multiple input sources\n",
    "* [`Gst-nvinfer`](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html) - NVIDIA TensorRT based plugin for primary and secondary (attribute classification of primary objects) detection and classification respectively\n",
    "* [`Gst-nvtracker`](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvtracker.html) - OpenCV based tracker plugin (Gst-nvtracker) for object tracking with unique ID\n",
    "* [`Gst-nvmultistreamtiler`](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvmultistreamtiler.html) - Multi Stream Tiler plugin for forming a 2D array of frames\n",
    "*  [`Gst-nvdsosd`](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvdsosd.html) - Onscreen Display (OSD) plugin  to draw shaded boxes, rectangles and text on the composited frame using the generated metadata\n",
    "* [`Gst-filesink`](https://gstreamer.freedesktop.org/documentation/coreelements/filesink.html?gi-language=python) - write stream data to a given file location\n",
    "\n",
    "The app accepts one or more H.264/H.265 video streams as input. It creates\n",
    "a source bin for each input and connects the bins to an instance of the\n",
    "\"nvstreammux\" element, which forms the batch of frames. The batch of\n",
    "frames is first fed to an \"nvinfer\" element, acting as the primary GPU inference engine (pgie), for batched inferencing. This is linked to an \"nvtracker\" instance which tracks the objects\n",
    "detected by the pgie, which is followed by three more instances of \"nvinfer\", each performing secondary classification as secondary GPU inference engines (sgie). The batched buffer is then\n",
    "composited into a 2D tile array using \"nvmultistreamtiler.\" \"nvdsosd\" draws the bounding boxes and text from the metadata generated by the \"nvinfer\" elements, and the final output stream is saved to a file by the filesink.\n",
    "\n",
    "**Note: a few additional elements are left out for simplicity, but will be described in detail in the application."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Probe Function\n",
    "Each nvinfer element generates and attaches metadata to the Gst Buffer. Metadata starts at the batch level, created by the Gst-nvstreammux plugin, and subsidiary metadata structures can hold frame metadata, object metadata, and display metadata. See [DeepStream documentation](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_metadata.html) for more details about metadata structure.\n",
    "\n",
    "This means that by attaching a probe function at the end of the pipeline, at the sink of the tiler, we can retrieve the buffer and extract meaningful information about the objects from these inferences and customize some display parameters, such as the bounding box color of these objects.\n",
    "\n",
    "Let's dive into some code!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 - Import necessary libraries\n",
    "\n",
    "First, we import necessary modules for this app. This of course, includes `pyds`, the module containing the DeepStream Python API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "import configparser\n",
    "import math\n",
    "\n",
    "import gi\n",
    "gi.require_version('Gst', '1.0')\n",
    "from gi.repository import GLib, Gst\n",
    "\n",
    "import pyds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - Declare class label IDs\n",
    "Define the class labels with their corresponding IDs according to the inference engine. To be used with an object counter in the probe function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PGIE_CLASS_ID_VEHICLE = 0\n",
    "PGIE_CLASS_ID_BICYCLE = 1\n",
    "PGIE_CLASS_ID_PERSON = 2\n",
    "PGIE_CLASS_ID_ROADSIGN = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Declare tiler properties and OSD properties\n",
    "These values will be used to set the width and height properties of the tiler, in pixels.\n",
    "Use CPU process mode for OSD and display text in the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TILED_OUTPUT_WIDTH=1920 # Tiler output width\n",
    "TILED_OUTPUT_HEIGHT=1080 # Tiler output height\n",
    "\n",
    "# NvOSD options\n",
    "OSD_PROCESS_MODE= 0 # Process mode for NvOSD plugin. 0 for CPU, 1 for GPU\n",
    "OSD_DISPLAY_TEXT= 1 # Enable/disable display of all text in NvOSD plugin. 0 for disable, 1 for enable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 - Set file paths and additional options\n",
    "Define the input stream file list, output file, and configuration files for GIEs and the tracker. We'll discuss the other options listed here in detail [later](#Simple-Customizations) in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# File paths\n",
    "input_file_list = [\"file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264\", \\\n",
    "                   \"file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264\", \\\n",
    "                   \"file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264\", \\\n",
    "                   \"file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264\"] \n",
    "                    # List of paths to input streams in URI format. Here we use 4 of the same stream as input.\n",
    "output_file = \"output.mp4\" # Output file location\n",
    "pgie_config_file = \"./configs/dslaunchpad_pgie_config.txt\" # Path to pgie config file\n",
    "tracker_config_file = \"./configs/dslaunchpad_tracker_config.txt\" # Path to tracker config file\n",
    "sgie1_config_file = \"./configs/dslaunchpad_sgie1_config.txt\" # Path to config file for first sgie\n",
    "sgie2_config_file = \"./configs/dslaunchpad_sgie2_config.txt\" # Path to config file for second sgie\n",
    "sgie3_config_file = \"./configs/dslaunchpad_sgie3_config.txt\" # Path to config file for third sgie\n",
    "\n",
    "# Tracker options\n",
    "enable_tracker = 1 # Enable/disable tracker and SGIEs. 0 for disable, 1 for enable\n",
    "\n",
    "# Note: for colors, each Red/Green/Blue/Alpha (RGBA) value should be between 0.0 and 1.0, inclusive.\n",
    "# Bounding box options\n",
    "bbox_border_color = {\"R\": 1.0, \"G\": 0.0, \"B\": 0.0, \"A\": 1.0} # Color of bounding box. Set to red\n",
    "bbox_has_bg_color = False # Bool for whether bounding box has background color\n",
    "bbox_bg_color = {\"R\": 1.0, \"G\": 0.0, \"B\": 0.0, \"A\": 0.2} # Color of bbox background. Set to red with transparency\n",
    "\n",
    "# Display text options, to be added to the frame.\n",
    "text_x_offset = 10 # Offset in the x direction where string should appear\n",
    "text_y_offset = 12 # Offset in the y direction where string should appear\n",
    "text_font_name = \"Serif\" # Font name\n",
    "text_font_size = 10 # Font size\n",
    "text_font_color = {\"R\" : 1.0, \"G\": 1.0, \"B\": 1.0, \"A\": 1.0} # Color of text font. Set to white\n",
    "text_set_bg_color = True # Bool for whether text box has background color\n",
    "text_bg_color = {\"R\": 0.0, \"G\": 0.0, \"B\": 0.0, \"A\": 1.0} # Color of text box background. Set to black"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5 - Define the probe function\n",
    "Let's write our probe function, to be attached at the sink pad of the tiler.\n",
    "\n",
    "This probe function will retrieve the Gst buffer, then retrieve the batch level metadata from the buffer. From the batch level metadata, we retrieve a list of frame metadata (one frame meta for each input stream), and from each frame metadata, we retrieve a list of object metadata, which were attached to the frame by the GIEs.\n",
    "\n",
    "For each object in a frame, we can apply customizations, such as setting the color of the bounding box and applying a background color to the bbox.\n",
    "\n",
    "For each frame, we can add text to the display by acquiring and attaching a display meta object. We set the text to be displayed and customize parameters such as the position of the text, the font, and the text box."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tiler_sink_pad_buffer_probe(pad,info,u_data):\n",
    "    frame_number=0\n",
    "    #Intiallizing object counter with 0.\n",
    "    obj_counter = {\n",
    "        PGIE_CLASS_ID_VEHICLE:0,\n",
    "        PGIE_CLASS_ID_PERSON:0,\n",
    "        PGIE_CLASS_ID_BICYCLE:0,\n",
    "        PGIE_CLASS_ID_ROADSIGN:0\n",
    "    }\n",
    "    num_rects=0\n",
    "    gst_buffer = info.get_buffer()\n",
    "    if not gst_buffer:\n",
    "        print(\"Unable to get GstBuffer \")\n",
    "        return\n",
    "\n",
    "    # Retrieve batch metadata from the gst_buffer\n",
    "    # Note that pyds.gst_buffer_get_nvds_batch_meta() expects the\n",
    "    # C address of gst_buffer as input, which is obtained with hash(gst_buffer)\n",
    "    batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))\n",
    "    l_frame = batch_meta.frame_meta_list\n",
    "    while l_frame is not None:\n",
    "        try:\n",
    "            # Note that l_frame.data needs a cast to pyds.NvDsFrameMeta\n",
    "            # The casting is done by pyds.NvDsFrameMeta.cast()\n",
    "            # The casting also keeps ownership of the underlying memory\n",
    "            # in the C code, so the Python garbage collector will leave\n",
    "            # it alone.\n",
    "            frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)\n",
    "        except StopIteration:\n",
    "            break\n",
    "\n",
    "        frame_number=frame_meta.frame_num\n",
    "        num_rects = frame_meta.num_obj_meta\n",
    "        l_obj=frame_meta.obj_meta_list\n",
    "        while l_obj is not None:\n",
    "            try:\n",
    "                # Casting l_obj.data to pyds.NvDsObjectMeta\n",
    "                obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)\n",
    "            except StopIteration:\n",
    "                break\n",
    "            obj_counter[obj_meta.class_id] += 1 # Increment object counter\n",
    "            rectparams = obj_meta.rect_params # Retrieve bounding box parameters\n",
    "            # Set color of bbox. (red, green, blue, alpha), 0.0 <= RGBA <= 1.0\n",
    "            rectparams.border_color.set(bbox_border_color[\"R\"], bbox_border_color[\"G\"], bbox_border_color[\"B\"], bbox_border_color[\"A\"])\n",
    "            # Set whether bbox has background color\n",
    "            rectparams.has_bg_color = bbox_has_bg_color\n",
    "            # If bbox has background color, set background color\n",
    "            if bbox_has_bg_color:\n",
    "                rectparams.bg_color.set(bbox_bg_color[\"R\"], bbox_bg_color[\"G\"], bbox_bg_color[\"B\"], bbox_bg_color[\"A\"])\n",
    "            try: \n",
    "                l_obj=l_obj.next\n",
    "            except StopIteration:\n",
    "                break\n",
    "\n",
    "        # Acquiring a display meta object. The memory ownership remains in\n",
    "        # the C code so downstream plugins can still access it. Otherwise\n",
    "        # the garbage collector will claim it when this probe function exits.\n",
    "        display_meta=pyds.nvds_acquire_display_meta_from_pool(batch_meta)\n",
    "        display_meta.num_labels = 1\n",
    "        py_nvosd_text_params = display_meta.text_params[0]\n",
    "        # Setting display text to be shown on screen\n",
    "        # Note that the pyds module allocates a buffer for the string, and the\n",
    "        # memory will not be claimed by the garbage collector.\n",
    "        # Reading the display_text field here will return the C address of the\n",
    "        # allocated string. Use pyds.get_string() to get the string content.\n",
    "        py_nvosd_text_params.display_text = \"Frame Number={} Number of Objects={} Vehicle_count={} Person_count={}\".format(frame_number, num_rects, obj_counter[PGIE_CLASS_ID_VEHICLE], obj_counter[PGIE_CLASS_ID_PERSON])\n",
    "\n",
    "        # Now set the offsets where the string should appear\n",
    "        py_nvosd_text_params.x_offset = text_x_offset\n",
    "        py_nvosd_text_params.y_offset = text_y_offset\n",
    "\n",
    "        # Font, font-color and font-size\n",
    "        py_nvosd_text_params.font_params.font_name = text_font_name\n",
    "        py_nvosd_text_params.font_params.font_size = text_font_size\n",
    "        # set(red, green, blue, alpha); set to White\n",
    "        py_nvosd_text_params.font_params.font_color.set(text_font_color[\"R\"], text_font_color[\"G\"], text_font_color[\"B\"], text_font_color[\"A\"])\n",
    "\n",
    "        # Text background color\n",
    "        py_nvosd_text_params.set_bg_clr = text_set_bg_color\n",
    "        # set(red, green, blue, alpha); set to Black\n",
    "        py_nvosd_text_params.text_bg_clr.set(text_bg_color[\"R\"], text_bg_color[\"G\"], text_bg_color[\"B\"], text_bg_color[\"A\"])\n",
    "        # Using pyds.get_string() to get display_text as string\n",
    "        print(pyds.get_string(py_nvosd_text_params.display_text))\n",
    "        # Add the display meta to the frame\n",
    "        pyds.nvds_add_display_meta_to_frame(frame_meta, display_meta)\n",
    "        try:\n",
    "            l_frame=l_frame.next\n",
    "        except StopIteration:\n",
    "            break\n",
    "    \n",
    "    return Gst.PadProbeReturn.OK\t\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6 - Define helper functions for creating source bins\n",
    "\n",
    "To allow for any video format, we use GStreamer's [uridecodebin](https://gstreamer.freedesktop.org/documentation/playback/uridecodebin.html?gi-language=python), which will automatically resolve the type of source element needed for the given URI and connect it to a decoder.\n",
    "\n",
    "Since there are properties we may want to set based on the type of source, we write callback functions to connect to certain signals. The `create_source_bin` helper function takes an index and a URI and creates a uridecodebin element, then connects the callback functions to the signals of interest.\n",
    "\n",
    "In this case, we have two callback functions. We connect `cb_newpad` to the \"pad-added\" signal, so that every time a pad is added (for sources with both audio and video, a pad is added for each), we check if the pad is for video, and if it is, we link the decodebin only if it has picked an nvidia decoder plugin. We connect `decodebin_child_added` to the \"child-added\" signal. When a child is added, if it is a [`decodebin`](https://gstreamer.freedesktop.org/documentation/playback/decodebin.html?gi-language=python) element (used internally by the uridecodebin), then we connect the callback to the \"child-added\" signal of that decodebin. If the child added was a source, then we check if the source has the `drop-on-latency` property and set to true if it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cb_newpad(decodebin, decoder_src_pad,data):\n",
    "    print(\"In cb_newpad\\n\")\n",
    "    caps=decoder_src_pad.get_current_caps()\n",
    "    if not caps:\n",
    "        caps = decoder_src_pad.query_caps()\n",
    "    gststruct=caps.get_structure(0)\n",
    "    gstname=gststruct.get_name()\n",
    "    source_bin=data\n",
    "    features=caps.get_features(0)\n",
    "\n",
    "    # Need to check if the pad created by the decodebin is for video and not\n",
    "    # audio.\n",
    "    print(\"gstname=\",gstname)\n",
    "    if(gstname.find(\"video\")!=-1):\n",
    "        # Link the decodebin pad only if decodebin has picked nvidia\n",
    "        # decoder plugin nvdec_*. We do this by checking if the pad caps contain\n",
    "        # NVMM memory features.\n",
    "        print(\"features=\",features)\n",
    "        if features.contains(\"memory:NVMM\"):\n",
    "            # Get the source bin ghost pad\n",
    "            bin_ghost_pad=source_bin.get_static_pad(\"src\")\n",
    "            if not bin_ghost_pad.set_target(decoder_src_pad):\n",
    "                sys.stderr.write(\"Failed to link decoder src pad to source bin ghost pad\\n\")\n",
    "        else:\n",
    "            sys.stderr.write(\" Error: Decodebin did not pick nvidia decoder plugin.\\n\")\n",
    "\n",
    "def decodebin_child_added(child_proxy,Object,name,user_data):\n",
    "    print(\"Decodebin child added:\", name, \"\\n\")\n",
    "    # Connect callback to internal decodebin signal\n",
    "    if(name.find(\"decodebin\") != -1):\n",
    "        Object.connect(\"child-added\",decodebin_child_added,user_data)\n",
    "\n",
    "    if \"source\" in name:\n",
    "        source_element = child_proxy.get_by_name(\"source\")\n",
    "        if source_element.find_property('drop-on-latency') != None:\n",
    "            Object.set_property(\"drop-on-latency\", True)\n",
    "\n",
    "def create_source_bin(index,uri):\n",
    "    print(\"Creating source bin\")\n",
    "\n",
    "    # Create a source GstBin to abstract this bin's content from the rest of the\n",
    "    # pipeline\n",
    "    bin_name=\"source-bin-%02d\" %index\n",
    "    print(bin_name)\n",
    "    nbin=Gst.Bin.new(bin_name)\n",
    "    if not nbin:\n",
    "        sys.stderr.write(\" Unable to create source bin \\n\")\n",
    "\n",
    "    # Source element for reading from the uri.\n",
    "    # We will use decodebin and let it figure out the container format of the\n",
    "    # stream and the codec and plug the appropriate demux and decode plugins.\n",
    "    uri_decode_bin=Gst.ElementFactory.make(\"uridecodebin\", \"uri-decode-bin\")\n",
    "    if not uri_decode_bin:\n",
    "        sys.stderr.write(\" Unable to create uri decode bin \\n\")\n",
    "    # We set the input uri to the source element\n",
    "    uri_decode_bin.set_property(\"uri\",uri)\n",
    "    # Connect to the \"pad-added\" signal of the decodebin which generates a\n",
    "    # callback once a new pad for raw data has been created by the decodebin\n",
    "    uri_decode_bin.connect(\"pad-added\",cb_newpad,nbin)\n",
    "    uri_decode_bin.connect(\"child-added\",decodebin_child_added,nbin)\n",
    "\n",
    "    # We need to create a ghost pad for the source bin which will act as a proxy\n",
    "    # for the video decoder src pad. The ghost pad will not have a target right\n",
    "    # now. Once the decode bin creates the video decoder and generates the\n",
    "    # cb_newpad callback, we will set the ghost pad target to the video decoder\n",
    "    # src pad.\n",
    "    Gst.Bin.add(nbin,uri_decode_bin)\n",
    "    bin_pad=nbin.add_pad(Gst.GhostPad.new_no_target(\"src\",Gst.PadDirection.SRC))\n",
    "    if not bin_pad:\n",
    "        sys.stderr.write(\" Failed to add ghost pad in source bin \\n\")\n",
    "        return None\n",
    "    return nbin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7 - Define bus call function\n",
    "Each pipeline contains a bus, which handles forwarding messages from streaming threads to the application. A message handler must be set, which will periodically check for new messages and call the callback function when a message is available. We define the callback below to handle end-of-stream (EOS) messages, warnings, and errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bus_call(bus, message, loop):\n",
    "    t = message.type\n",
    "    if t == Gst.MessageType.EOS:\n",
    "        sys.stdout.write(\"End-of-stream\\n\")\n",
    "        loop.quit()\n",
    "    elif t==Gst.MessageType.WARNING:\n",
    "        err, debug = message.parse_warning()\n",
    "        sys.stderr.write(\"Warning: %s: %s\\n\" % (err, debug))\n",
    "    elif t == Gst.MessageType.ERROR:\n",
    "        err, debug = message.parse_error()\n",
    "        sys.stderr.write(\"Error: %s: %s\\n\" % (err, debug))\n",
    "        loop.quit()\n",
    "    return True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8 - Build and run pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8-1 - Initialize GStreamer\n",
    "Here, we begin construction of the pipeline. We start by following the standard GStreamer initialization procedure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard initialization procedure\n",
    "Gst.init(None)\n",
    "\n",
    "# Create gstreamer elements\n",
    "# Create Pipeline element that will form a connection of other elements\n",
    "print(\"Creating Pipeline \\n \")\n",
    "pipeline = Gst.Pipeline()\n",
    "\n",
    "if not pipeline:\n",
    "    sys.stderr.write(\" Unable to create Pipeline \\n\")\n",
    "    \n",
    "number_sources = len(input_file_list) # Declare number of input sources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8-2 - Create pipeline elements\n",
    "We create each pipeline element, set their respective properties, and add them to the pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 8-2-1 - Create the streammux\n",
    "\n",
    "The Gst-nvstreammux plugin combines `batch-size` number of frames from multiple input sources into a batched buffer. This batch is pushed downstream when the batch is filled or the `batched-push-timeout` time limit is reached. All frames in the batch are scaled to the same resolution, specified by the `width` and `height` properties in pixels. See the DeepStream [documentation](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvstreammux.html) for more details and other properties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Creating streammux \\n \")\n",
    "\n",
    "# Create nvstreammux instance to form batches from one or more sources.\n",
    "streammux = Gst.ElementFactory.make(\"nvstreammux\", \"Stream-muxer\")\n",
    "if not streammux:\n",
    "    sys.stderr.write(\" Unable to create NvStreamMux \\n\")\n",
    "\n",
    "# Set streammux properties\n",
    "streammux.set_property('width', 1920)\n",
    "streammux.set_property('height', 1080)\n",
    "streammux.set_property('batch-size', number_sources)\n",
    "streammux.set_property('batched-push-timeout', 4000000)\n",
    "\n",
    "pipeline.add(streammux) # Add streammux to the pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 8-2-2 - Create a source bin for each input file\n",
    "We call our previously defined helper function for creating a source bin for each input file, and add it to the pipeline. The sink pad of the streammux is retrieved, and each source is linked to a sink pad of the streammux."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Iterate through each input source\n",
    "for i in range(number_sources):\n",
    "    print(\"Creating source_bin \",i,\" \\n \")\n",
    "    uri_name=input_file_list[i] # Retrieve input file URI\n",
    "    if uri_name.find(\"rtsp://\") == 0 :\n",
    "        is_live = True\n",
    "    source_bin=create_source_bin(i, uri_name) # Call helper to create source bin\n",
    "    if not source_bin:\n",
    "        sys.stderr.write(\"Unable to create source bin \\n\")\n",
    "    pipeline.add(source_bin) # Add source bin to pipeline\n",
    "    padname=\"sink_%u\" %i\n",
    "    sinkpad= streammux.get_request_pad(padname) # Retrieve a sink pad from the streammux element\n",
    "    if not sinkpad:\n",
    "        sys.stderr.write(\"Unable to create sink pad bin \\n\")\n",
    "    srcpad=source_bin.get_static_pad(\"src\") # Retrieve the source pad of the source bin\n",
    "    if not srcpad:\n",
    "        sys.stderr.write(\"Unable to create src pad bin \\n\")\n",
    "    srcpad.link(sinkpad) # Link the source pad of the source bin to the sink pad of the streammux"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 8-2-3 - Create nvinfer element as PGIE\n",
    "The Gst-nvinfer plugin takes the batched buffer from the streammux and performs inference using NVIDIA TensorRT. We create this element to perform inference in primary mode. Behaviour of the inference are set through properties in the configuration file. In this case, we are using ResNet10 to detect objects and classify them as Person, Vehicle, Bicycle, or Roadsign."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Creating nvinfer (PGIE) \\n \")\n",
    "\n",
    "# Create nvinfer element\n",
    "pgie = Gst.ElementFactory.make(\"nvinfer\", \"primary-inference\")\n",
    "if not pgie:\n",
    "    sys.stderr.write(\" Unable to create pgie \\n\")\n",
    "    \n",
    "# Set nvinfer properties\n",
    "pgie.set_property('config-file-path', pgie_config_file)\n",
    "pgie_batch_size=pgie.get_property(\"batch-size\")\n",
    "\n",
    "if(pgie_batch_size != number_sources):\n",
    "    print(\"WARNING: Overriding infer-config batch-size\",pgie_batch_size,\" with number of sources \", number_sources,\" \\n\")\n",
    "    pgie.set_property(\"batch-size\",number_sources)\n",
    "\n",
    "\n",
    "# Add pgie to pipeline\n",
    "pipeline.add(pgie)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a quick look at the contents of the pgie configuration file. These configurations determine the behavior of the plugin, such as the model used for inference, the process mode (primary or secondary), etc. See the DeepStream [documentation](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html#gst-nvinfer-file-configuration-specifications) for more details on file configuration specifications.\n",
    "```bash\n",
    "[property]\n",
    "gpu-id=0 # Device ID of GPU to use for inference\n",
    "net-scale-factor=0.0039215697906911373 # Pixel scaling factor\n",
    "model-file=../../../../samples/models/Primary_Detector/resnet10.caffemodel # Path to caffemodel file\n",
    "proto-file=../../../../samples/models/Primary_Detector/resnet10.prototxt # Path to prototxt file\n",
    "# Path toserialized model engine file\n",
    "model-engine-file=../../../../samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine\n",
    "labelfile-path=../../../../samples/models/Primary_Detector/labels.txt # Path to text file containing labels\n",
    "int8-calib-file=../../../../samples/models/Primary_Detector/cal_trt.bin # Path to calibration file\n",
    "force-implicit-batch-dim=1 # Force the implicit batch dimension mode\n",
    "batch-size=1 # Number of frames/objects to be inferred together in a batch\n",
    "network-mode=1 # Data format to be used by inference - 0: FP32, 1: INT8, 2: FP16\n",
    "num-detected-classes=4 # Number of classes detected by the network\n",
    "interval=0 # Number of consecutive batches to be skipped for inference\n",
    "gie-unique-id=1 # Unique ID to assign to GIE\n",
    "output-blob-names=conv2d_bbox;conv2d_cov/Sigmoid # Array of output layer names\n",
    "#scaling-filter=0\n",
    "#scaling-compute-hw=0\n",
    "\n",
    "[class-attrs-all]\n",
    "pre-cluster-threshold=0.2 # Detection threshold to be applied prior to clustering\n",
    "eps=0.2 # Epsilon values for OpenCV grouprectangles() function and DBSCAN algorithm\n",
    "group-threshold=1 # Threshold value for rectangle merging for OpenCV grouprectangles() function\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 8-2-4 - Create tracker, if enabled\n",
    "The Gst-nvtracker plugin uses a low-level tracker library (NvDCF in our case) to track the objects detected by the PGIE with persistent IDs over time. Properties for the tracker are also set through a configuration file. See the DeepStream [documentation](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvtracker.html) for more details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if enable_tracker:\n",
    "    print(\"Creating nvtracker\\n \")\n",
    "    \n",
    "    # Create nvtracker element\n",
    "    tracker = Gst.ElementFactory.make(\"nvtracker\", \"tracker\")\n",
    "    if not tracker:\n",
    "        sys.stderr.write(\" Unable to create tracker \\n\")\n",
    "\n",
    "    # Parse tracker config file and set properties\n",
    "    config = configparser.ConfigParser()\n",
    "    config.read(tracker_config_file)\n",
    "    config.sections()\n",
    "\n",
    "    for key in config['tracker']:\n",
    "        if key == 'tracker-width' :\n",
    "            tracker_width = config.getint('tracker', key)\n",
    "            tracker.set_property('tracker-width', tracker_width)\n",
    "        if key == 'tracker-height' :\n",
    "            tracker_height = config.getint('tracker', key)\n",
    "            tracker.set_property('tracker-height', tracker_height)\n",
    "        if key == 'gpu-id' :\n",
    "            tracker_gpu_id = config.getint('tracker', key)\n",
    "            tracker.set_property('gpu_id', tracker_gpu_id)\n",
    "        if key == 'll-lib-file' :\n",
    "            tracker_ll_lib_file = config.get('tracker', key)\n",
    "            tracker.set_property('ll-lib-file', tracker_ll_lib_file)\n",
    "        if key == 'll-config-file' :\n",
    "            tracker_ll_config_file = config.get('tracker', key)\n",
    "            tracker.set_property('ll-config-file', tracker_ll_config_file)\n",
    "\n",
    "    # Add tracker to pipeline\n",
    "    pipeline.add(tracker)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 8-2-5 - Create SGIEs, if tracker enabled\n",
    "If the tracker is enabled, we create three more Gst-nvinfer elements to perform inference in secondary mode. These SGIEs will perform inference on the tracked objects from the upstream nvtracker element and classify each vehicle based on car make, car color, and car type, respectively to each SGIE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if enable_tracker:\n",
    "    print(\"Creating nvinfer (SGIE1) \\n \")\n",
    "    \n",
    "    # Create nvinfer element\n",
    "    sgie1 = Gst.ElementFactory.make(\"nvinfer\", \"secondary1-nvinference-engine\")\n",
    "    if not sgie1:\n",
    "        sys.stderr.write(\" Unable to make sgie1 \\n\")\n",
    "\n",
    "    print(\"Creating nvinfer (SGIE2) \\n \")\n",
    "    \n",
    "    # Create nvinfer element\n",
    "    sgie2 = Gst.ElementFactory.make(\"nvinfer\", \"secondary2-nvinference-engine\")\n",
    "    if not sgie2:\n",
    "        sys.stderr.write(\" Unable to make sgie2 \\n\")\n",
    "        \n",
    "    print(\"Creating nvinfer (SGIE3) \\n \")\n",
    "    \n",
    "    # Create nvinfer element\n",
    "    sgie3 = Gst.ElementFactory.make(\"nvinfer\", \"secondary3-nvinference-engine\")\n",
    "    if not sgie3:\n",
    "        sys.stderr.write(\" Unable to make sgie3 \\n\")\n",
    "\n",
    "    # Set config file for each SGIE\n",
    "    sgie1.set_property('config-file-path', sgie1_config_file)\n",
    "    sgie2.set_property('config-file-path', sgie2_config_file)\n",
    "    sgie3.set_property('config-file-path', sgie3_config_file)\n",
    "\n",
    "    # Add SGIEs to pipeline\n",
    "    pipeline.add(sgie1)\n",
    "    pipeline.add(sgie2)\n",
    "    pipeline.add(sgie3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 8-2-6 - Create multistreamtiler\n",
    "The Gst-nvmultistreamtiler plugin takes the batched buffer from upstream and composites the streams into a `rows` x `columns` 2D tile in row-major order based on stream IDs. The 2D tile will have `width` x `height` resolution. See the DeepStream [documentation](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvmultistreamtiler.html) for details and additional properties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Creating tiler \\n \")\n",
    "\n",
    "# Create nvmultistreamtiler element\n",
    "tiler=Gst.ElementFactory.make(\"nvmultistreamtiler\", \"nvtiler\")\n",
    "if not tiler:\n",
    "    sys.stderr.write(\" Unable to create tiler \\n\")\n",
    "\n",
    "# Calculate number of rows and columns based on number of input sources\n",
    "tiler_rows=int(math.sqrt(number_sources))\n",
    "tiler_columns=int(math.ceil((1.0*number_sources)/tiler_rows))\n",
    "\n",
    "# Set tiler properties\n",
    "tiler.set_property(\"rows\",tiler_rows)\n",
    "tiler.set_property(\"columns\",tiler_columns)\n",
    "tiler.set_property(\"width\", TILED_OUTPUT_WIDTH)\n",
    "tiler.set_property(\"height\", TILED_OUTPUT_HEIGHT)\n",
    "\n",
    "# Add tiler to pipeline\n",
    "pipeline.add(tiler)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 8-2-7 - Create nvvideoconvert to convert from NV12 to RGBA as required by nvdsosd\n",
    "The Gst-nvvideoconvert plugin performs video color format conversion. The nvdsosd plugin, next in the pipeline, requires RGBA format, which nvvideoconvert plugin outputs. See the DeepStream [documentation](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvvideoconvert.html) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Creating nvvidconv \\n \")\n",
    "\n",
    "# Create nvvideoconvert element\n",
    "nvvidconv = Gst.ElementFactory.make(\"nvvideoconvert\", \"convertor\")\n",
    "if not nvvidconv:\n",
    "    sys.stderr.write(\" Unable to create nvvidconv \\n\")\n",
    "\n",
    "# Add nvvideoconvert to pipeline\n",
    "pipeline.add(nvvidconv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 8-2-8 - Create OSD to draw on the converted RGBA buffer\n",
    "The Gst-nvdsosd plugin accepts an RGBA buffer from upstream and draws the boudning boxes and text given the attached metadata. The text and bounding box parameters are configurable through the metadata, which we customize in the probe function. The `process-mode` property indicates whether CPU mode or GPU mode is used to the draw the objects, and the `display-text` property indicates whether or not to display all text. See the DeepStream [documentation](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvdsosd.html) for more details and properties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Creating nvosd \\n \")\n",
    "\n",
    "# Create nvdsosd element\n",
    "nvosd = Gst.ElementFactory.make(\"nvdsosd\", \"onscreendisplay\")\n",
    "if not nvosd:\n",
    "    sys.stderr.write(\" Unable to create nvosd \\n\")\n",
    "    \n",
    "# Set nvosd properties\n",
    "nvosd.set_property('process-mode',OSD_PROCESS_MODE)\n",
    "nvosd.set_property('display-text',OSD_DISPLAY_TEXT)\n",
    "\n",
    "# Add nvosd to pipeline\n",
    "pipeline.add(nvosd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 8-2-9 - Create set of nvvidconv, encoder, mux, and parser to prepare stream for filesink\n",
    "The nvvideoconvert element will convert the buffer to the color format accepted by the nvv4l2h264enc element, which will encode the buffer into h264 video format. The [qtmux](https://gstreamer.freedesktop.org/documentation/isomp4/qtmux.html?gi-language=python) will merge any audio into the video before passing the buffer to the [h264parse](https://gstreamer.freedesktop.org/documentation/videoparsersbad/h264parse.html?gi-language=python), which will parse the recently encoded h264 stream to prepare for writing to the filesink."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Creating nvvidconv \\n \")\n",
    "\n",
    "# Create nvvideoconvert element\n",
    "nvvidconv2 = Gst.ElementFactory.make(\"nvvideoconvert\", \"convertor2\")\n",
    "if not nvvidconv2:\n",
    "    sys.stderr.write(\" Unable to create nvvidconv2 \\n\")\n",
    "\n",
    "print(\"Creating nvv4l2h264enc \\n \")\n",
    "\n",
    "# Create nvv4l2h264enc element\n",
    "encoder = Gst.ElementFactory.make(\"nvv4l2h264enc\", \"encoder\")\n",
    "if not encoder:\n",
    "    sys.stderr.write(\" Unable to create encoder \\n\")\n",
    "\n",
    "print(\"Creating qtmux \\n \")\n",
    "\n",
    "# Create qtmux element\n",
    "mux = Gst.ElementFactory.make(\"qtmux\", \"muxer\")\n",
    "if not mux:\n",
    "    sys.stderr.write(\" Unable to create muxer \\n\")\n",
    "\n",
    "# Create h264parse element\n",
    "print(\"Creating h264parse\\n \")\n",
    "parser1 = Gst.ElementFactory.make(\"h264parse\", \"h264-parser2\")\n",
    "if not parser1:\n",
    "    sys.stderr.write(\" Unable to create parser1 \\n\")\n",
    "\n",
    "# Add all to pipeline\n",
    "pipeline.add(nvvidconv2)\n",
    "pipeline.add(encoder)\n",
    "pipeline.add(mux)\n",
    "pipeline.add(parser1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 8-2-10 - Create filesink to save output stream to file\n",
    "The filesink element writes the buffer to a file given the `location`. See the Gstreamer [documentation](https://gstreamer.freedesktop.org/documentation/coreelements/filesink.html?gi-language=python) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Creating FileSink \\n\")\n",
    "\n",
    "# Create filesink\n",
    "sink = Gst.ElementFactory.make(\"filesink\", \"nvvideo-renderer\")\n",
    "\n",
    "# Set filesink properties\n",
    "sink.set_property('location', output_file)\n",
    "sink.set_property(\"sync\",0) # Don't sync to clock, to allow samples to be played as fast as possible\n",
    "\n",
    "# Add sink to pipeline\n",
    "pipeline.add(sink)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8-3 - Link the pipeline elements\n",
    "After we construct the elements and add them to the pipeline, they must be linked to each other in the proper order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note that the sources were already linked to the streammux\n",
    "# Link streammux > pgie\n",
    "streammux.link(pgie)\n",
    "# If tracker is enabled, link pgie > tracker > sgie1 > sgie2 > sgie3 > tiler\n",
    "if enable_tracker:\n",
    "    pgie.link(tracker)\n",
    "    tracker.link(sgie1)\n",
    "    sgie1.link(sgie2)\n",
    "    sgie2.link(sgie3)\n",
    "    sgie3.link(tiler)\n",
    "# Otherwise, linke pgie > tiler\n",
    "else:\n",
    "    pgie.link(tiler)\n",
    "# Link tiler > nvvidconv > nvosd > nvvidconv2 > encoder > parser1 > mux > sink\n",
    "tiler.link(nvvidconv)\n",
    "nvvidconv.link(nvosd)\n",
    "nvosd.link(nvvidconv2)\n",
    "nvvidconv2.link(encoder)\n",
    "encoder.link(parser1)\n",
    "parser1.link(mux)\n",
    "mux.link(sink)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8-4 - Create an event loop and bus messages to it\n",
    "We first create a mainloop, which continuously checks for occurrences on its watch. We add the bus signal to the watch and connect the previously defined callback function to the loop. This way, when a message is available on the bus, the callback function will be called to handle the message."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loop = GLib.MainLoop() # Create a mainloop\n",
    "bus = pipeline.get_bus() # Retrieve the bus from the pipeline\n",
    "bus.add_signal_watch() # Add a watch for new messages on the bus\n",
    "bus.connect (\"message\", bus_call, loop) # Connect the loop to the callback function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8-5 - Attach the probe function to extract the meta data generated\n",
    "We want the probe to be attached at an element after all GIEs to be able access the attached metadata, but before the buffer passes through the tiler, since it combines all frames. So we attach the probe to the sink of the tiler."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We add probe to the sink pad of the tiler element, since by that time, \n",
    "# the buffer would have had gotten all the metadata.\n",
    "\n",
    "# Retrieve the sink pad of the tiler\n",
    "tilersinkpad = tiler.get_static_pad(\"sink\")\n",
    "if not tilersinkpad:\n",
    "    sys.stderr.write(\" Unable to get sink pad of tiler \\n\")\n",
    "\n",
    "# Add probe function to the sink pad of the tiler\n",
    "tilersinkpad.add_probe(Gst.PadProbeType.BUFFER, tiler_sink_pad_buffer_probe, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8-6 - Run the pipline\n",
    "Note: this step will take several minutes to finish running. Do not execute the next cell until this step is complete, i.e. you see the text `<enum GST_STATE_CHANGE_SUCCESS of type Gst.StateChangeReturn>` after a series of `Frame Number = ... Number of Objects = ... Vehicle_count = ... Person_count = ...` prints and `End-of-stream` is reached."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(\"Starting pipeline \\n\")\n",
    "pipeline.set_state(Gst.State.PLAYING)\n",
    "try:\n",
    "    loop.run()\n",
    "except:\n",
    "    pass\n",
    "# cleaning up as the pipeline comes to an end\n",
    "pipeline.set_state(Gst.State.NULL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 9 - View the output written by filesink"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observe the output below. In the output video, you should see four streams corresponding to each input source tiled in a 2 x 2 matrix. At the top left of each tiled stream, there should be display text descrbing the frame number, number of objects, vehicle count, and person count for each frame. A red bounding box should appear around each detected object, along with a text label describing the object (Person, Car, etc.), and the car labels should also include information from the SGIEs, i.e. car make, type, and color.\n",
    "\n",
    "**Note: if the video does not fit into the Jupyter Notebook cell, you can scroll to the right in the cell and click the fullscreen button to view the video in its full resolution. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import Video\n",
    "\n",
    "Video(\"output.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Congratulations! \n",
    "You've run your first Deepstream Python app. Now that you've seen the pipeline code broken down, let's put it into a function so we can run it easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_and_run_pipeline():\n",
    "    # Standard initialization procedure\n",
    "    Gst.init(None)\n",
    "\n",
    "    # Create gstreamer elements\n",
    "    # Create Pipeline element that will form a connection of other elements\n",
    "    print(\"Creating Pipeline \\n \")\n",
    "    pipeline = Gst.Pipeline()\n",
    "\n",
    "    if not pipeline:\n",
    "        sys.stderr.write(\" Unable to create Pipeline \\n\")\n",
    "\n",
    "    number_sources = len(input_file_list) # Declare number of input sources\n",
    "    \n",
    "    print(\"Creating streammux \\n \")\n",
    "\n",
    "    # Create nvstreammux instance to form batches from one or more sources.\n",
    "    streammux = Gst.ElementFactory.make(\"nvstreammux\", \"Stream-muxer\")\n",
    "    if not streammux:\n",
    "        sys.stderr.write(\" Unable to create NvStreamMux \\n\")\n",
    "\n",
    "    # Set streammux properties\n",
    "    streammux.set_property('width', 1920)\n",
    "    streammux.set_property('height', 1080)\n",
    "    streammux.set_property('batch-size', number_sources)\n",
    "    streammux.set_property('batched-push-timeout', 4000000)\n",
    "\n",
    "    pipeline.add(streammux) # Add streammux to the pipeline\n",
    "    \n",
    "    # Iterate through each input source\n",
    "    for i in range(number_sources):\n",
    "        print(\"Creating source_bin \",i,\" \\n \")\n",
    "        uri_name=input_file_list[i] # Retrieve input file URI\n",
    "        if uri_name.find(\"rtsp://\") == 0 :\n",
    "            is_live = True\n",
    "        source_bin=create_source_bin(i, uri_name) # Call helper to create source bin\n",
    "        if not source_bin:\n",
    "            sys.stderr.write(\"Unable to create source bin \\n\")\n",
    "        pipeline.add(source_bin) # Add source bin to pipeline\n",
    "        padname=\"sink_%u\" %i\n",
    "        sinkpad= streammux.get_request_pad(padname) # Retrieve a sink pad from the streammux element\n",
    "        if not sinkpad:\n",
    "            sys.stderr.write(\"Unable to create sink pad bin \\n\")\n",
    "        srcpad=source_bin.get_static_pad(\"src\") # Retrieve the source pad of the source bin\n",
    "        if not srcpad:\n",
    "            sys.stderr.write(\"Unable to create src pad bin \\n\")\n",
    "        srcpad.link(sinkpad) # Link the source pad of the source bin to the sink pad of the streammux\n",
    "        \n",
    "        print(\"Creating nvinfer (PGIE) \\n \")\n",
    "\n",
    "    # Create nvinfer element\n",
    "    pgie = Gst.ElementFactory.make(\"nvinfer\", \"primary-inference\")\n",
    "    if not pgie:\n",
    "        sys.stderr.write(\" Unable to create pgie \\n\")\n",
    "\n",
    "    # Set nvinfer properties\n",
    "    pgie.set_property('config-file-path', pgie_config_file)\n",
    "    pgie_batch_size=pgie.get_property(\"batch-size\")\n",
    "\n",
    "    if(pgie_batch_size != number_sources):\n",
    "        print(\"WARNING: Overriding infer-config batch-size\",pgie_batch_size,\" with number of sources \", number_sources,\" \\n\")\n",
    "        pgie.set_property(\"batch-size\",number_sources)\n",
    "\n",
    "\n",
    "    # Add pgie to pipeline\n",
    "    pipeline.add(pgie)\n",
    "    \n",
    "    if enable_tracker:\n",
    "        print(\"Creating nvtracker\\n \")\n",
    "\n",
    "        # Create nvtracker element\n",
    "        tracker = Gst.ElementFactory.make(\"nvtracker\", \"tracker\")\n",
    "        if not tracker:\n",
    "            sys.stderr.write(\" Unable to create tracker \\n\")\n",
    "\n",
    "        # Parse tracker config file and set properties\n",
    "        config = configparser.ConfigParser()\n",
    "        config.read(tracker_config_file)\n",
    "        config.sections()\n",
    "\n",
    "        for key in config['tracker']:\n",
    "            if key == 'tracker-width' :\n",
    "                tracker_width = config.getint('tracker', key)\n",
    "                tracker.set_property('tracker-width', tracker_width)\n",
    "            if key == 'tracker-height' :\n",
    "                tracker_height = config.getint('tracker', key)\n",
    "                tracker.set_property('tracker-height', tracker_height)\n",
    "            if key == 'gpu-id' :\n",
    "                tracker_gpu_id = config.getint('tracker', key)\n",
    "                tracker.set_property('gpu_id', tracker_gpu_id)\n",
    "            if key == 'll-lib-file' :\n",
    "                tracker_ll_lib_file = config.get('tracker', key)\n",
    "                tracker.set_property('ll-lib-file', tracker_ll_lib_file)\n",
    "            if key == 'll-config-file' :\n",
    "                tracker_ll_config_file = config.get('tracker', key)\n",
    "                tracker.set_property('ll-config-file', tracker_ll_config_file)\n",
    "\n",
    "        # Add tracker to pipeline\n",
    "        pipeline.add(tracker)\n",
    "        \n",
    "        print(\"Creating nvinfer (SGIE1) \\n \")\n",
    "    \n",
    "        # Create nvinfer element\n",
    "        sgie1 = Gst.ElementFactory.make(\"nvinfer\", \"secondary1-nvinference-engine\")\n",
    "        if not sgie1:\n",
    "            sys.stderr.write(\" Unable to make sgie1 \\n\")\n",
    "\n",
    "        print(\"Creating nvinfer (SGIE2) \\n \")\n",
    "\n",
    "        # Create nvinfer element\n",
    "        sgie2 = Gst.ElementFactory.make(\"nvinfer\", \"secondary2-nvinference-engine\")\n",
    "        if not sgie2:\n",
    "            sys.stderr.write(\" Unable to make sgie2 \\n\")\n",
    "\n",
    "        print(\"Creating nvinfer (SGIE3) \\n \")\n",
    "\n",
    "        # Create nvinfer element\n",
    "        sgie3 = Gst.ElementFactory.make(\"nvinfer\", \"secondary3-nvinference-engine\")\n",
    "        if not sgie3:\n",
    "            sys.stderr.write(\" Unable to make sgie3 \\n\")\n",
    "\n",
    "        # Set config file for each SGIE\n",
    "        sgie1.set_property('config-file-path', sgie1_config_file)\n",
    "        sgie2.set_property('config-file-path', sgie2_config_file)\n",
    "        sgie3.set_property('config-file-path', sgie3_config_file)\n",
    "\n",
    "        # Add SGIEs to pipeline\n",
    "        pipeline.add(sgie1)\n",
    "        pipeline.add(sgie2)\n",
    "        pipeline.add(sgie3)\n",
    "        \n",
    "    print(\"Creating tiler \\n \")\n",
    "\n",
    "    # Create nvmultistreamtiler element\n",
    "    tiler=Gst.ElementFactory.make(\"nvmultistreamtiler\", \"nvtiler\")\n",
    "    if not tiler:\n",
    "        sys.stderr.write(\" Unable to create tiler \\n\")\n",
    "\n",
    "    # Calculate number of rows and columns based on number of input sources\n",
    "    tiler_rows=int(math.sqrt(number_sources))\n",
    "    tiler_columns=int(math.ceil((1.0*number_sources)/tiler_rows))\n",
    "\n",
    "    # Set tiler properties\n",
    "    tiler.set_property(\"rows\",tiler_rows)\n",
    "    tiler.set_property(\"columns\",tiler_columns)\n",
    "    tiler.set_property(\"width\", TILED_OUTPUT_WIDTH)\n",
    "    tiler.set_property(\"height\", TILED_OUTPUT_HEIGHT)\n",
    "\n",
    "    # Add tiler to pipeline\n",
    "    pipeline.add(tiler)\n",
    "    \n",
    "    print(\"Creating nvvidconv \\n \")\n",
    "\n",
    "    # Create nvvideoconvert element\n",
    "    nvvidconv = Gst.ElementFactory.make(\"nvvideoconvert\", \"convertor\")\n",
    "    if not nvvidconv:\n",
    "        sys.stderr.write(\" Unable to create nvvidconv \\n\")\n",
    "\n",
    "    # Add nvvideoconvert to pipeline\n",
    "    pipeline.add(nvvidconv)\n",
    "    \n",
    "    print(\"Creating nvosd \\n \")\n",
    "\n",
    "    # Create nvdsosd element\n",
    "    nvosd = Gst.ElementFactory.make(\"nvdsosd\", \"onscreendisplay\")\n",
    "    if not nvosd:\n",
    "        sys.stderr.write(\" Unable to create nvosd \\n\")\n",
    "\n",
    "    # Set nvosd properties\n",
    "    nvosd.set_property('process-mode',OSD_PROCESS_MODE)\n",
    "    nvosd.set_property('display-text',OSD_DISPLAY_TEXT)\n",
    "\n",
    "    # Add nvosd to pipeline\n",
    "    pipeline.add(nvosd)\n",
    "    \n",
    "    print(\"Creating nvvidconv \\n \")\n",
    "\n",
    "    # Create nvvideoconvert element\n",
    "    nvvidconv2 = Gst.ElementFactory.make(\"nvvideoconvert\", \"convertor2\")\n",
    "    if not nvvidconv2:\n",
    "        sys.stderr.write(\" Unable to create nvvidconv2 \\n\")\n",
    "\n",
    "    print(\"Creating nvv4l2h264enc \\n \")\n",
    "\n",
    "    # Create nvv4l2h264enc element\n",
    "    encoder = Gst.ElementFactory.make(\"nvv4l2h264enc\", \"encoder\")\n",
    "    if not encoder:\n",
    "        sys.stderr.write(\" Unable to create encoder \\n\")\n",
    "\n",
    "    print(\"Creating qtmux \\n \")\n",
    "\n",
    "    # Create qtmux element\n",
    "    mux = Gst.ElementFactory.make(\"qtmux\", \"muxer\")\n",
    "    if not mux:\n",
    "        sys.stderr.write(\" Unable to create muxer \\n\")\n",
    "\n",
    "    # Create h264parse element\n",
    "    print(\"Creating h264parse\\n \")\n",
    "    parser1 = Gst.ElementFactory.make(\"h264parse\", \"h264-parser2\")\n",
    "    if not parser1:\n",
    "        sys.stderr.write(\" Unable to create parser1 \\n\")\n",
    "\n",
    "    # Add all to pipeline\n",
    "    pipeline.add(nvvidconv2)\n",
    "    pipeline.add(encoder)\n",
    "    pipeline.add(mux)\n",
    "    pipeline.add(parser1)\n",
    "    \n",
    "    print(\"Creating FileSink \\n\")\n",
    "\n",
    "    # Create filesink\n",
    "    sink = Gst.ElementFactory.make(\"filesink\", \"nvvideo-renderer\")\n",
    "\n",
    "    # Set filesink properties\n",
    "    sink.set_property('location', output_file)\n",
    "    sink.set_property(\"sync\",0)\n",
    "\n",
    "    # Add sink to pipeline\n",
    "    pipeline.add(sink)\n",
    "    \n",
    "    # Note that the sources were already linked to the streammux\n",
    "    # Link streammux > pgie\n",
    "    streammux.link(pgie)\n",
    "    # If tracker is enabled, link pgie > tracker > sgie1 > sgie2 > sgie3 > tiler\n",
    "    if enable_tracker:\n",
    "        pgie.link(tracker)\n",
    "        tracker.link(sgie1)\n",
    "        sgie1.link(sgie2)\n",
    "        sgie2.link(sgie3)\n",
    "        sgie3.link(tiler)\n",
    "    # Otherwise, linke pgie > tiler\n",
    "    else:\n",
    "        pgie.link(tiler)\n",
    "    # Link tiler > nvvidconv > nvosd > nvvidconv2 > encoder > parser1 > mux > sink\n",
    "    tiler.link(nvvidconv)\n",
    "    nvvidconv.link(nvosd)\n",
    "    nvosd.link(nvvidconv2)\n",
    "    nvvidconv2.link(encoder)\n",
    "    encoder.link(parser1)\n",
    "    parser1.link(mux)\n",
    "    mux.link(sink)\n",
    "    \n",
    "    loop = GLib.MainLoop() # Create a mainloop\n",
    "    bus = pipeline.get_bus() # Retrieve the bus from the pipeline\n",
    "    bus.add_signal_watch() # Add a watch for new messages on the bus\n",
    "    bus.connect (\"message\", bus_call, loop) # Connect the loop to the callback function\n",
    "    \n",
    "    # We add probe to the sink pad of the tiler element, since by that time, \n",
    "    # the buffer would have had gotten all the metadata.\n",
    "\n",
    "    # Retrieve the sink pad of the tiler\n",
    "    tilersinkpad = tiler.get_static_pad(\"sink\")\n",
    "    if not tilersinkpad:\n",
    "        sys.stderr.write(\" Unable to get sink pad of tiler \\n\")\n",
    "\n",
    "    # Add probe function to the sink pad of the tiler\n",
    "    tilersinkpad.add_probe(Gst.PadProbeType.BUFFER, tiler_sink_pad_buffer_probe, 0)\n",
    "    \n",
    "    print(\"Starting pipeline \\n\")\n",
    "    pipeline.set_state(Gst.State.PLAYING)\n",
    "    try:\n",
    "        loop.run()\n",
    "    except:\n",
    "        pass\n",
    "    # cleaning up as the pipeline comes to an end\n",
    "    pipeline.set_state(Gst.State.NULL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you can simply build and run the pipeline by calling this function (after picking a new output file name). Again, do not run the cell after until the pipeline runs to completion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_file = \"output2.mp4\"\n",
    "build_and_run_pipeline()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View the newly written file output. This should be identical to the first output video."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "Video(\"output2.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Customizations\n",
    "Let's talk about a few simple ways to customize your new DeepStream Python application. Recall the file paths and additional options in [Step 4](#Step-4---Set-file-paths-and-additional-options)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change input sources\n",
    "You can add any number of URIs to our input file list, and as you've seen, the application will create a source bin for each  input. The batch size properties for the streammux and the pgie will be set according to the number of sources, and the number of rows and columns for the tiler will be calculated and set accordingly. Note that the tiler will shrink/stretch each stream for the 2D tile to fit the total resolution.\n",
    "\n",
    "Let's try a file list of 2 URIs.\n",
    "\n",
    "Note: these URIs do not need to be the same as each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_file_list = [\"file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264\", \\\n",
    "                   \"file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264\"]\n",
    "                    # List of paths to input streams in URI format\n",
    "output_file = \"output3.mp4\" # Output file location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "build_and_run_pipeline()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observe the new tiled output with 2 streams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "Video(\"output3.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Disable the tracker and SGIEs\n",
    "Through the `enable_tracker` setting, we can disable the tracker by setting the value to `0`, or `False`. Pipeline creation will then skip the creation/addition/linkage of the tracker and 3 SGIE elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_tracker = 0  #Enable/disable tracker and SGIEs. 0 for disable, 1 for enable\n",
    "output_file = \"output4.mp4\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "build_and_run_pipeline()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observe that SGIE inference no longer occurs, so information previously inferred from SGIEs (car make, color, type) are no longer generated and attached to the buffer, and thus do not appear in the OSD output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Video(\"output4.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bounding box options\n",
    "The probe function extracts the object level metadata attached to the buffer and can set the parameters of the bounding boxes by retrieving the NvOSDRectParams structure of the object meta. For example, the color of the border of the bounding box, whether or not the bounding box has a background color, and what that background color is. Originally, the bbox border was set to red, and there was no background color.\n",
    "\n",
    "Let's see what happens when we set the border to blue, enable a background color, and set the background color to transparent red."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note: for colors, each Red/Green/Blue/Alpha (RGBA) value should be between 0.0 and 1.0, inclusive.\n",
    "# Bounding box options\n",
    "bbox_border_color = {\"R\": 0.0, \"G\": 0.0, \"B\": 1.0, \"A\": 1.0} # Color of bounding box. Set to blue\n",
    "bbox_has_bg_color = True # Bool for whether bounding box has background color\n",
    "bbox_bg_color = {\"R\": 1.0, \"G\": 0.0, \"B\": 0.0, \"A\": 0.2} # Color of bbox background. Set to red with transparency"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "output_file = \"output5.mp4\"\n",
    "build_and_run_pipeline()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observe the pretty new colors on our bounding boxes!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "Video(\"output5.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display text options\n",
    "The display text refers to the text currently at the top left of each frame, with the content \"Frame Number= ... Number of Objects = ...\". This is created and attached to the frame meta by the probe function. We can change the x/y offset (position of the text with respect to the frame), the font of the text, font size, font color, whether the text box has a background, and the color of that background.\n",
    "\n",
    "Let's move the text to the right, try a different, larger font, make the text green, and remove the background of the text box."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display text options, to be added to the frame.\n",
    "text_x_offset = 800 # Offset in the x direction where string should appear\n",
    "text_y_offset = 15 # Offset in the y direction where string should appear\n",
    "text_font_name = \"Sans\" # Font name\n",
    "text_font_size = 16 # Font size\n",
    "text_font_color = {\"R\" : 0.0, \"G\": 1.0, \"B\": 0.0, \"A\": 1.0} # Color of text font. Set to green\n",
    "text_set_bg_color = False # Bool for whether text box has background color\n",
    "text_bg_color = {\"R\": 0.0, \"G\": 0.0, \"B\": 0.0, \"A\": 1.0} # Color of text box background. Set to black"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before running the pipeline, change our input file list to use only one stream for clarity. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_file_list = [\"file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264\"] # List of paths to input streams in URI format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "output_file = \"output6.mp4\"\n",
    "build_and_run_pipeline()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "Video(\"output6.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Try it yourself!\n",
    "Here are all of the options in one place again. Go ahead and make some changes, then run the pipeline and see how they affect the output!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# File paths\n",
    "input_file_list = [\"file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264\", \\\n",
    "                   \"file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264\"] \n",
    "                    # List of paths to input streams in URI format\n",
    "output_file = \"output7.mp4\" # Output file location\n",
    "\n",
    "# Tracker options\n",
    "enable_tracker = 1 # Enable/disable tracker and SGIEs. 0 for disable, 1 for enable\n",
    "\n",
    "# Note: for colors, each Red/Green/Blue/Alpha (RGBA) value should be between 0.0 and 1.0, inclusive.\n",
    "# Bounding box options\n",
    "bbox_border_color = {\"R\": 1.0, \"G\": 0.0, \"B\": 0.0, \"A\": 1.0} # Color of bounding box. Set to red\n",
    "bbox_has_bg_color = False # Bool for whether bounding box has background color\n",
    "bbox_bg_color = {\"R\": 1.0, \"G\": 0.0, \"B\": 0.0, \"A\": 0.2} # Color of bbox background. Set to red with transparency\n",
    "\n",
    "# Display text options, to be added to the frame.\n",
    "text_x_offset = 10 # Offset in the x direction where string should appear\n",
    "text_y_offset = 12 # Offset in the y direction where string should appear\n",
    "text_font_name = \"Serif\" # Font name\n",
    "text_font_size = 10 # Font size\n",
    "text_font_color = {\"R\" : 1.0, \"G\": 1.0, \"B\": 1.0, \"A\": 1.0} # Color of text font. Set to white\n",
    "text_set_bg_color = True # Bool for whether text box has background color\n",
    "text_bg_color = {\"R\": 0.0, \"G\": 0.0, \"B\": 0.0, \"A\": 1.0} # Color of text box background. Set to black"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "build_and_run_pipeline()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Video(output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations on completing your first DeepStream Python application! Please check out our [deepstream_python-apps](https://github.com/NVIDIA-AI-IOT/deepstream_python_apps) repository. There, you can explore some other use cases through our sample applications, and introduce yourself to the publicly available Python bindings code."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
